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* Abstract , »* ■ 

• ' \ 

■ The retrieval decision problem is corisi(^red from the viewpoint of 

' " ' ''\ \ 

a decision theory approach. A threshol^v ruler based on earlier rules 

for indexing decisions is considered arid "^nalyzedj^pr* retrieval decisions 

as'a measure of retrieval performance. 'Thd threshold rule %s seen as a . 

good descriptive desigji measure of what a reksdnable retrieval system 

should jDe ^ble to do. A retrieval raecfianism o^ randomly drawing docu-^ 

ments is aiialv2e to determine the relative strength )ii the threshold 

V 

rule. The Neyman-Pearson-'rule is shown to be a better a priori decision, 
r^ile for retrieval; attempting to mascimi^e precisio^i^ subject to a fixed ' 
level o;f recall, instead of setting a lower limit upoVprecision, as - 

. dQ.e.s„tJ:]te.. t;iu:e^bpld.xuie^„ Jlhe'jLhreshQld rule is sgen a.q\ a nftcfic;<?/^ryj 

but not sufficient, condition for effective retrieval. Penally, ^ ' 
sufficient condition for tiaie threshold rule illustrates thk relation- 



^ship between it and the Neyman-Pearson rule. 
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Introduction ^ 

Consider the decision 'of whether or not ta retrieve a given\iocument 
in response to a specific user query for infojrma^ion. Tbis approa^ to 
-information retrieval systems has been dii^cussed in detail previously 
[l, 2, 3]. The basic model is illustrated* below in Figure' 1. Here one^s^ 
can se& the lalternatives: to retrieve or not to retrieve; the 'states-of- 
nature: relevant o^r nonrelevant; and the evaluations: * * 
Y, = tbe value o^- retrieving a relevant 'document ^ 
-C = the cost of retrieving, a nonrelevant document, 
-C^ = the cost of not retrieving a relevant document, 



and 



= the value of hot 'retrieving a nonrelevant document. 



We ^hall assume that value always exceeds cost, i.e.,\ 

^ ' 

^1 "^S/^^' ^2'^^lX^' ^l'^^2^^' "^l^^Q^ ^' ^^^^^ seems intuitively t^}- 

-p-——- ; ^ ^ ^ . _ . ^ ^ ' 

Assuming that .the case is of decision mafting under risk, one has 
G^= generality = f»r (relevant) j with G £ (0,1). It has been shown [2j 
th^t.one^ retrieves ^ document if and on/y if E(A^) = V^G+(-C^) {l-G)yE{A^) 
1 ' N =^ (-C^)G'^V^ (1-G) or / ^ . 

.G>(V2+C^)/(V^+C^+C2+V2). . ' ^ ' (X) 

I In^rder to be^more generally \isefuL consider a retrieval system that 
assignT^o each document a value of 'a random Vai:i||y^ Z, called a retrieval 
status value, that attempts to estimate the ddciiiment's potential relevance 
to a* specific user queyry ^23. We can nod define^:^ . ' 
P(Z) = Pr (relevance/2) , ' ; ^ • 

f^^Z) =^Pr (Z/relevance; r *" . 

f-(Z>^= Pr(Z/nonrelevance) , » " . • * ; ' 

, * . ' • . ' \ . ' ' 

f (Z) Pr(Z) = f rz)G+f^(Z) (1-G) , and ^1 ' ' • 

L(Z) = f^(Z)/f2(Z) = likelihood ratio.. ^ j- 
It can be. shown* ^2"]that the decision rule dictating]' retrieval can now be restated 



6 . . • 




' P(z) 



f ^ - / 

^ s Thie; all owe; /f 



>(V2+o()/(v/+C>C2W2). 
L(Z)>'(V^j^) (1-G)/ (V^^^G = K. ' . (2) 



This allows /for the definition of 



R = cange of *Z for retrieval =^{z|l(Z)>k}. 



/ 

Evalttati^n Mechanism 

/ ■ % 

TWe retrieval system'^an be eraluated in terws of its effectiveness 

7 . .. • ; ' ■ - . . . 

in retrieving relevant documents. Consider the traditioi^l performance 



= recall = Pr (retrieve Ireldvai^ce) = * Z f (Z) , 
= fcillout = Pr( retrieve J* nai\relevance)= £ f ^ (Z) , and 

- precisdonr ^ Pri^reievancij retrieve) • , ' 

= ReG/PF (retrieve) = ReG/ (ReG+Fa (1-G) ) . , * 

The Neyman-Pearson cj?iterion \2j is *' 

Max' (1-Fa) s.t. (1-Re) =oC, ^ 

which is equivalent to maximizing precision 5ul)ject to a fixed level (l-o() of recall'. 

Tjie Threshold Rule ^ . ^ . , 

] - * *\ 

% Ma'ron [4] has suggested a rule for indexing defcis ions that an analbg ihay have 

• . . ! * ' ^ . 

applicability here for retrieval decisions. In our terms for the retrievaj. 

decision problem, we have a4 the analog'U>f Maron's ru^le, the staiement^ 

The act of retrieval should ^idd information about the relevance of 

each document. , This translates to the rule that givep a feet af 

users wijth similar queries far information, that would be shown a 

given document, that document should be retrieved if,t>he pr9portion^ 

of* the users that would find the dbcument relevant exceeds the 

probability^ that any document selected at random ;woui^d be relevant • ' 

to the query. % . y 

' .... ♦ 




In terms of thfe retrieval decision model /.the rule implies that 
Pr(relevdnce I retrieve) - Pn^ PrTi?elevance) = G.^ 



(3) 



This formula can be manipulated algebraically to yield* 



- Re = Pr (retrieve! relevance)^ Pr (retrlevej, = ReG+Fa (1-X3) , * • (3a), 

• ^ ' , ^ J 

f' ^ ' . Pn = Pr(relevance| retrieve) >Tr(rerevAnceJ not retrieve) . _ '(3b) 

•* ' = (l-Re)G/(l-ReG-Fa(l-G) J , and 

. ! " * * ' 

O ' . t5f^-(Z)> Fa =If2(z).. ■ • ■ , -^^^^^ 



Note^hat this threshold rule' does not seem' so much an a priori decision 
rule to determine, whether or not to retrieve a giv^n document i^ response tff - 
d^^sp&Afic UBex^c^r^^JRath&iCr is is an af terrthe-fact descriptive design 
ineasure to e^l^late the^overall retrieval mechanism, based 6n a vej^y rigorous., 
and common ^sense appifoac'h. • 

Random Retrieval * , ' - * / 

Consider the situation of randomly drawing a subset -o^ the documents in 

the collection ai^ an 'act of retrieval. This seemingly poor method of retrieval 

should provide a baseline for con^aring retriisval decision rules. .Let 

N = number of documents in' the collection,' 

R = number of re,levant<doc|uments in the collectior^ 

\ % f 

n = number ^of documents randomly retr>ieved, and 

f 

r = number of relevant docoments randomly retrieve.d, 
we shall 'assume' that N, R, and |i are fixed and krtown^. This, incidentally, is 
at variance with Cooper^ / who assumes that'Ni, R, and t are fixed, in a 

modeTlthat .considers minimizing (n-r) ; the search length. It is ' noteworthy 

— t f 

that this is equivalent to maximjlzing precision for a fixed level of recall [^23. 

ri; our situation, we have: 
G = R/N,^ 

Re = r/R, , , * 

Fa = (n-r) /.(N-R) , and 
. Pn ='*r/n. • 
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In the situation of a random retrieval mechanism, r becomes a random variable 

having a hypergeometric prpbeOiility distribution; i.e.. 

This means that - / ' ' * ' ' ' 

u = E(r) = nR/N « nG. " . (4) 



8 



* 



z • ■ • ' " 

rule, 16 now wrijbten as: 
, " Pn = t/n> G = R/N, or • ^ ' ' 



:\ The threshold 



>jiG = u. 



(5) 



'4* ' 



» Since the hypergeoi^etric .distribution is symmetric around its mean, the* 



* probability that a random draw will satisfy the tljreSnold rule,. which- is 
the equivalent to the probability that a hypergeometrxc random variable 

exceeds its mean, its approximately equal *to one-half, Th\i4, the ^thres- ^ 

, * ' * i . * ' ^. * - ' 

hold rule is not* an especially strong discriminat;^ng decis;Lon rule fot ' 
retsrieval, in that even a rahdom draw of documents can satisfy it neatly 
- half the time. This implies that tiie threshold rule is a minimum re- 

qi;iremen^/^r retrieval systems,* ^ necessary But not sufficient condition 

f ^ ^ ' 

for effec?tive retrieval. • . ^ 

. ? 

Suppose we assume «that n'^R. Further, suppose that for each of the 
n documents retrieved, • tfier^ is a constatTt probability p of the document 
^ being relevant. Then, "r has a binominal probability distribx^tion, i.e., 
(r) (^"P?"!""^ ' 0,i,2,...,n^ and ^ 



.E^(r) = np. _ . . , ^ 

. ^ 1 ^ - ' \ * ^ ' 

P0r the system to be more ^f f ective • tha o a mei^ random draw, we would 

require . , » • ' • 

, - r 

E^(r>^=''np)^u = nG, or 



p>G. 



/ 



(6) 



Note that^ ♦ . 

, p =,Pr(relevancej retrieve) = Pn, • (7) • 

*so that formulA *<6) is equivalent to fopnula (3), the threstiold rule^. 

The threshold rule implies thai? a ^retrieval system should be atJle to 

generate a relevant document with greater frequency than a random' draw, 
I . 

which is eminently reasonable. ' ^ 



The Neytnan-pegrson Rule ^ , , ; - / ' ' ^ 

The Neyman- Pear son lemma of statistical 'decision^ theory can t>e used 
^2^^ to show that the Neyman-Pearsoh criterion,, discusse'd above, can be * 

# ' I ' " - - - 

optimized by retrieving all documents with a retrieval Status value Z in 
• the range ^ ^ ^ ^ 

R = L(Z)y'K]. , " - (8) 

.'1 ' ' • ' * 

K IS now|determined by the formula . . , ▼ 



Re =Z f, (Z) = 1- < . - ♦ 



(9) 



Let us recall that the threshold rule can be stated aa 

/ - ' . .; • • : ' \ ' 

Re.=rf (Z)> Fa = ^ f (&) . ' * • ■ . ^^^^ , 

We can nOw derive a sufficient, but riot -necessary, condition' for the 
threshold' rule. If ' i \ ^ ' ■ - * - * , 

* * • 1 » • 

r*(z) = f^(z)/f2 (z) y k'^^i ^ziR, -.1 (10) 

then the .threshold rule will hoid. Equating the constant K of formula 
•(8) to that of formula (2) yields: . •' ^ 

■K = (-Vj^+C ) (1-G)/(V =C^)G,>-1, or * ' „ . 

'g < [V^+C^)/{V^+C^+C^+V^)\ 4 . (11) 

Note that formula (11) is' just the opposite result of formula, (1). What 
this says is .ttiat for small enough G, the Neyman-Pearson rule will satisfy 
the th^-eahold rule. This secerns reasonable, in light of^^rmula -(3). For 
large G, the^TWi4pation is dependent ixpon the.vflue of ^ and the specific 
values of 'f^(Z), an^ t^(Z) for the\ various values of Z ift. In poi-nt of " 
fact, it would be a most perverse retrieval system that would sartisfy 
the NeymarilPearson r,ule but riot allow the^ threshold rule to be sajtisfied 
for a "reasona±»le" level of'^ecall. The real difference here is^hat the \ 

Neymaji-Pearson rule maximi^s precision subject to a fixed lev^l of- recall, 

- ^' • . • * ' t ' ^ 

whi-le the threshold rul^ imppses a minimum level on precision. ^ 



Suramary and Goocl^ions . » * ' * • . 

A threshold rule has 'b^en analyzed as a useful toQl to evaluate 
information .retrieval -systems. It provides a goqd descriptive design 
measure ,of what. a reasonable retrieval mechanism should ^be able 
•accomplish. Yet, the threshold rul^ is not seen ^s an especially 
strong discriminating rule foy a priori ••retrieval decisions, ^ince' — 

the rule can, be gatisfiod riearJLy half the "time by a random draW'.* 

i * ' ^» ^ - » / 

' # - * • ' ' ' ' I * 

The threshold rule can be ar^^lyze'd irt t^rmte of the retrieval deci-- 

*' ' ' . ' ' * '• 

' sion m^'del, derived" from J^e assumptions for the*bi»mial pro6dbility 

distribution case of the random draw'. When compa^'ed to the Neymc^- 

Pearson rule, the threshold rule is not seen as having %he ability ' 

.tc\^' compute the extent to which precision^can be itf^imized.* The \ y 

NeVman-Pearson rule is shbWn to be a better a priori decision rule 

for retrieval. Moreover, the* Neyman-Pearson rule is a suFficienf, 

condition for the .threshold rule, for appropriate values of gen^eralit^. 

Thus the threShoid rule provides .a necessary, but not sufficient, 

measiyje of the leasit a retrieval system should be able to achieve. 
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